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Scoring Mechanism Selection Along Multiple Dimensions 



RELATED APPLICATIONS AND PRIORITY CLAIM 

[0001] This patent application is related to U.S. Patent Application No. 10/XXX,XXX 
(Attorney Docket No. 50269-0565), filed on February 26, 2004, entitled "Associating 
Product Offerings with Product Abstractions" naming Brian Acton, Brian Stolte, Rob 
Solomon, Jeff Weiner, Catherine Cadogan, Matt Heist, Manish Baldua, and Charles Arendt 
as inventors, the entire contents of these prior applications are hereby incorporated by 
reference in their entirety for all purposes. 

[0002] This patent application is related to U.S. Patent Application No. 10/XXX,XXX 
(Attorney Docket No. 50269-0566), filed on February 26, 2004, entitled "Product Data 
Classification" naming Brian Acton, Brian Stolte, Rob Solomon, Jeff Weiner, Catherine 
Cadogan, Matt Heist, Manish Baldua, and Charles Arendt as inventors, the entire contents of 
these prior applications are hereby incorporated by reference in their entirety for all purposes. 

FIELD OF THE INVENTION 

[0003] The present invention relates to query processing. The invention specifically 
relates to selecting a scoring mechanism from a plurality scoring mechanisms for processing 
queries. 

BACKGROUND OF THE INVENTION 

[0004] A product search mechanism is a mechanism that, in response to a user query, 
provides search results that identify one or more products that are selected based on the 
query. A user query is typically a Boolean logic phrase contained in a string. 
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[0005] Product search mechanisms are often associated with specific websites. For 
example, a website may provide to a user's browser a web page that includes a field for 
specifying a query. The user enters a query into the field, and then selects a control that 
causes the query to be transmitted back to a product search mechanism associated with the 
website. The product search mechanism then processes the query, and sends a web page that 
contains the search results back to the user's browser. 

[0006] In product search mechanisms, results are generated by query processing engines. 
Query processing engines generally include one or more scoring mechanisms. A scoring 
mechanism is a mechanism that employs a particular algorithm for finding matching results 
for a user query from among a pool of candidate results. The act of using a scoring 
mechanism to determine the search results for a query is referred to herein as "scoring the 
query" with the scoring mechanism. 

[0007] Different scoring mechanisms may use different techniques for determining how 
to score a query. The algorithm used by a scoring mechanism may, for example, select the 
matching results based on a comparison between key words in the query and key words in 
product descriptions. Such a scoring mechanism is referred to herein as a "keyword- 
matching" scoring mechanism. 

[0008] For example, consider a keyword-matching scoring mechanism used by a product 
search engine. A user may want to find all products that have the words "Hess AND 
Siddhartha AND book". The keyword-matching scoring mechanism would search the 
product database and, by comparing the keys words in the query against key words in 
product descriptions, provide a result containing a reference to a copy of Herman Hess' book 
"Siddhartha". 
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[0009] Some systems use a single scoring mechanism for scoring a user query, as was the 
case for the example above. A problem with this approach is that a single scoring 
mechanism may not produce acceptable results for all user queries. For example, if a system 
uses only a keyword-matching scoring mechanism, then the results for some product 
searches may be acceptable, such as the Siddhartha search described above. However, other 
searches for other products may not produce acceptable results, such as a search for a laptop 
based on the query: "laptop AND 2.3 GHz AND nVidia". Using a keyword-matching 
scoring mechanism, a laptop with a 2.4 GHz processor with an nVidia graphics card would 
not be selected by the scoring mechanism as one of the results. 

[0010] Another type of scoring mechanism is a Bayesian scoring mechanism. Bayesian 
scoring mechanisms use Bayesian classifiers, which are neural networks trained on sample 
data. The sample data contains sample user queries and appropriate result sets for those 
queries. After training on the sample data, a Bayesian classifier provides an estimate of how 
likely a given candidate item from the pool of product information is to match a user query. 
[0011] Another approach to producing results for a user query is to (1) use multiple 
Bayesian scoring mechanisms in the system, (2) determine the "relevance" of the search 
results produced by each scoring mechanism, and (3) provide to the user that submitted the 
query the results from the scoring mechanism with the maximum relevance for that query. 
The relevance for a Bayesian scoring mechanism is determined as a functional composition 
of the likelihood that each result in the set of results produced by the scoring mechanism 
matches the user query. 

[0012] In the relevance maximization approach, the search results will often include only 
unsponsored candidate items, where an unsponsored item is one for which no party 
associated with an item is charged a fee for inclusion of the item in a result set. This 
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approach aides the user, because the user receives the most relevant result set from among 
those generated by the multiple scoring mechanisms. However, such an approach does not 
take into consideration the revenue generation needs of the company providing the search 
results. A result set maximizing the relevance and exclusively consisting of unsponsored 
links generates no revenue. 

[0013] Even if sponsored results are considered during the scoring process, the relevance 
maximization approach still has a problem in that it leads to a sacrifice of generated revenue 
in order to maximize relevance of the search results. For example, consider the choice 
between two scoring mechanisms: scoring mechanism A which has a relevance score, using 
some relevance scoring system, of 85% and potential revenue of $20 and a scoring 
mechanism B which has a relevance score of 86% and a potential revenue of $1 . Using this 
approach would cause the search provider to forego $19 while providing the customer with 
only marginally better search results. 

[0014] Another approach to producing results for a user query is the revenue 
maximization approach. This approach uses multiple scoring mechanisms for each user 
query and selects the one for which the maximum revenue would be generated given its 
produced result set. Such an approach aides the search provider by providing revenue for the 
service it provides, but has a problem in that it leads to a sacrifice of relevance of the result 
set to the user query. For example, consider the choice between two scoring mechanisms: 
scoring mechanism A which has a relevance score of 85% and potential revenue of $20 and a 
scoring mechanism B which has a relevance score of 12% and a potential revenue of $21 . 
Simply maximizing on revenue would cause the search provider to choose to use a scoring 
mechanism with very low relevance over a scoring mechanism with high relevance in order 
to make $1 more. 
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[0015] Clearly, there is a need for techniques that select from multiple scoring 
mechanisms but still avoid choosing a scoring mechanism based on a single parameter value. 
[0016] The approaches described in this section are approaches that may be pursued, but 
not necessarily approaches that have been previously conceived or pursued. Therefore, 
unless otherwise indicated, it should not be assumed that any of the approaches described in 
this section qualify as prior art merely by virtue of their inclusion in this section. 
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BRIEF DESCRIPTION OF THE DRAWINGS 

[0017] The present invention is illustrated by way of example, and not by way of 
limitation, in the figures of the accompanying drawings and in which like reference numerals 
refer to similar elements and in which: 

[0018] FIG. 1 is a block diagram that depicts example architectural components and 
layout for a system for providing search results according to one embodiment of the 
invention 

[0019] FIG. 2 is a flow diagram that illustrates a process for choosing a scoring 
mechanism based on a user query. 

[0020] FIG. 3 is a flow diagram that illustrates a process for calculating user retention 
based on cookies. 

[0021] FIG. 4 is a flow diagram that illustrates a process for calculating user retention 
based on user history. 

[0022] FIG. 5 is a block diagram that illustrates a computer system upon which an 
embodiment of the invention may be implemented. 
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DETAILED DESCRIPTION OF THE INVENTION 

[0023] Techniques for selecting a scoring mechanism from a plurality scoring 
mechanisms for processing queries are described. In the following description, for the 
purposes of explanation, numerous specific details are set forth in order to provide a 
thorough understanding of the present invention. It will be apparent, however, that the 
present invention may be practiced without these specific details. In other instances, well- 
known structures and devices are shown in block diagram form in order to avoid 
unnecessarily obscuring the present invention. 

GENERAL OVERVIEW 
[0024] Techniques are provided for picking a particular scoring mechanism from a 
plurality of scoring mechanisms for a particular query. Consider a search mechanism that 
has multiple scoring mechanisms at its disposal - each implemented as a separate subroutine 
in the search mechanism. Depending on the query that comes in, different scoring 
mechanisms may produce better results than the others. The techniques described hereafter 
determine which of a plurality of scoring mechanisms will produce the "best" results for a 
particular query, where "best" is determined by a variety of factors. 

DEFINITIONS 

[0025] The following terms shall be used hereafter to describe embodiments of the 
invention: 

[0026] Scoring mechanism: any technique to produce query results based on a user 
query. 

[0027] Product offering: an offer to sell a particular product by a particular party. 
[0028] Product abstraction: abstract representations of a product. 
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[0029] Product item: any product-related item about which a user may want to perform a 
search. E.g. a merchant, product offering, or product abstraction. 
[0030] Sponsored product item: a product item for which a party associated with the 
particular product item will or has been charged a fee for each inclusion of the particular 
product item in results provided by the search mechanism. 

[0031] User retention for a scoring mechanism: likelihood that a user that issued a query 
will find the results from the scoring mechanism useful. 



IDENTIFYING BETTER RESULTS 
[0032] In order to produce "better results", the search mechanism determines parameter 
values for each scoring mechanism, given the query submitted by the user. For example, the 
search mechanism may determine, for each scoring mechanism, a percentage of sponsored 
product items in the result sets, a user retention parameter value, and a relevance of results 
produced parameter value. The search mechanism performs a functional composition of the 
parameter values in order to determine, for the particular query at issue, which scoring 
mechanisms will produce the best results. 

[0033] Consider this example, a search mechanism chooses among three scoring 
mechanisms, A, B, and C for processing a query for a copy for Herman Hess' book 
"Siddhartha" ("Siddhartha query"). The search mechanism determines the parameter values 
for relevance, percentage of sponsored product items, and user retention for each of the three 
scoring mechanisms. For the purpose of illustration, it shall be assumed that each scoring 
mechanism has a strong point. Specifically, scoring mechanism A has a relevance score of 
90%, a percentage of sponsored product items of 60%, and a user retention value of 50%. 
Scoring mechanism B has a relevance score of 60%, a percentage of sponsored product items 
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of 70%, and a user retention value of 90%. Scoring mechanism C has a relevance score of 
50%, a percentage of sponsored product items of 90%, and a user retention value of 70%. 
Depending on the functional composition used, any among these scoring mechanisms may 
have the highest total score and thereby be chosen. In this example, the product of the 
parameter values is calculated and scoring mechanism B has the highest score (378,000) 
followed by C (315,000) and A (270,000). Therefore, scoring mechanism B is chosen to 
provide results to the user who submitted the Siddhartha query. 

SYSTEM OVERVIEW 
[0034] FIG. 1 is a block diagram that depicts example architectural components and 
layout for a system for providing search results according to one embodiment of the 
invention. Techniques are described hereafter for selecting a scoring mechanism from a 
plurality of scoring mechanisms based on multiple dimensions. These techniques enable a 
broader system which may include, but neither must include nor is limited to the following 
components: 

[0035] one or more users 110, communicatively coupled to a network 120, that submit 
queries to a search mechanism 130 and receive results based on those queries; 
[0036] the network 120 for coupling the users 1 10 and the search mechanism 130; 
[0037] the search mechanism 130 for selecting a scoring mechanism from a plurality of 
scoring mechanisms based on multiple dimensions; 

[0038] a database 140, communicatively coupled to the search mechanism 130, for 
storing information related to the multiple scoring mechanisms and the products which the 
users are searching; and 
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[0039] a log 150, communicatively coupled to the search mechanism 130, for storing 
information related to the execution of the search mechanism and selection of scoring 
mechanisms. 

[0040] In a system 100, users 110 submit queries to a search mechanism 130 over a 
network. Users 110 may be human users, automated processes, or any other appropriate 
users. The network 120 may be a wireless network, dial up access, the Internet, a local area 
network (LAN), or any another appropriate communication network. The search mechanism 
130 may be a process running on one or more processors on a single computer, multiple 
processes running on one or more processors on a single computer, or multiple processes 
running on two or more computers. The users may submit their queries using file transport 
protocol (FTP), hypertext transport protocol (HTTP), secure HTTP (HTTPS), rsync, or 
another data transport protocol which is able to deliver data to the search mechanism. Any 
appropriate query type may be used, including Boolean queries and keyword searches. The 
queries may be submitted using any appropriate format, including Hypertext Markup 
Language (HTML), Extensible Markup Language (XML), character-delimited format, or 
fixed-column-width formats. 

[0041] The search mechanism 130 processes the query and provides results from a 
particular scoring mechanism. The particular scoring mechanism used to produce the results 
may be selected based on parameter values generated, based on the query, for multiple 
candidate scoring mechanisms. The techniques for doing this are described in greater detail 
in the Functional Overview. 

[0042] The search mechanism 130, in generating results for a user 110 query, obtains, 
from the database 140, information related to the candidate scoring mechanisms, such as the 
user retention values for each scoring mechanism. In this example, the database 140 is a 
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relational database. The database 140 may also be a flat file, Lightweight Directory Access 
Protocol (LDAP) directory, or another appropriate storage mechanism. 
[0043] While generating results for a query, the search mechanism 130 stores 
information related to result generation in a log file 150 . This stored information may 
include a record of which scoring mechanism was picked to provide results for which user 
110. The log file is a flat file, but may also be a relational database, a directory service, a 
LDAP directory, or another appropriate storage mechanism. In other embodiments, a single 
database 140 or log file 150 may perform the functions herein performed by the separate 
database 140 and log file 150. 

FUNCTIONAL OVERVIEW 
[0044] One of the operations performed by the system described herein is choosing a 
scoring mechanism, from among multiple candidate scoring mechanisms, based on a query 
submitted by a user 110. 

[0045] FIG. 2 is a flow diagram that illustrates a process for choosing a scoring 
mechanism from among a plurality of scoring mechanisms based on a user query. Referring 
to FIG. 2, a user query is received in step 210. For example, a user 110 submits the 
Siddhartha query via HTTP in HTML by typing the query into a webpage provided by the 
website associated with the search mechanism 130. As noted above, the query may be 
submitted in any appropriate format via any appropriate communication means. 
[0046] After receiving a user query in step 210, candidate search results are generated for 
each of a plurality of scoring mechanisms in step 220. For example, the search mechanism 
130 generates results for the Siddhartha query for both scoring mechanism A and scoring 
mechanism B. Results will generally consist of a set of links to one or more product items. 
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The results may include XML, HTML, or other appropriate links to any product item for 
which a user may want to search. 

[0047] According to one embodiment, the results for each scoring mechanism are 
generated by searching for matching query results in the database 140 using the matching 
algorithm associated with the scoring mechanism. The techniques for searching for matching 
items is based on the scoring mechanism being used, and may include key word matching, 
Bayesian classification, and/or matching based on product identifiers, such as Universal 
Product Code (UPC), International Standard Book Number (ISBN), manufacturer, 
manufacturer's part number, model number, or other appropriate identifiers. Bayesian 
classifiers are trained neural networks that provide an estimate for the likelihood that a 
particular product offering satisfies a query. More details about Bayesian classifiers may be 
found in David D. Lewis. "Naive (Bayes) at forty: The independence assumption in 
information retrieval". In Claire N'edellec and Celine Rouveirol, editors, Proceedings of 
ECML-98, 10th European Conference on Machine Learning, number 1398, pages 4-15, 
Chemnitz, DE, 1998. Springer Verlag, Heidelberg, DE. 

[0048] In step 230, for each scoring mechanism of the plurality of scoring mechanisms, 
multiple parameter values are generated. Some or all of the parameter values that are 
generated for each scoring mechanism may be based on the search results that the scoring 
mechanism would provide for the particular user query. For example, (1) a relevance score, 
(2) a percentage of sponsored product items score, and (3) a user retention score are 
generated for each of scoring mechanisms A, B, and C based on the results that the scoring 
mechanism would generate for the Siddhartha query. Other parameter values may also be 
used. Examples of how each of these parameters may be calculated shall be given hereafter. 
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[0049] After determining parameter values for each scoring mechanism in step 230, a 
scoring mechanism is chosen based on those parameter values (step 240). In one 
embodiment, the choice of scoring mechanism is based on the product of the multiple 
parameter values. For example, consider the example of scoring mechanisms A, B, and C 
with parameter values described in General Overview. Multiplying the parameter values for 
each scoring mechanism together, we determine that scoring mechanism B has the highest 
score and therefore choose scoring mechanism B. The determination of which scoring 
mechanism to choose may also be based on performing any appropriate mathematical 
functions on the parameter values, including a sum of parameter values; a product of squares 
of parameter values; a sum of squares of the parameter values; or a polynomial composition 
of the parameter values. 

[0050] In the example given above, a single scoring mechanism is selected for providing 
results to the query. However, in alternative embodiment, a subset of the scoring 
mechanisms may be chosen, where the search result returned to the user include a mixture of 
some or all of the results of each of the selected scoring mechanisms. 
[0051] The process described herein for choosing a scoring mechanism from among a 
plurality of scoring mechanisms based on a user query enables, among other things, a search 
mechanism to produce results that balance many aspects, represented by parameter values, of 
various scoring mechanisms in order to pick the best among them. 

DETERMINING RELEVANCE OF SCORING MECHANISM 
[0052] As mentioned above, a relevance parameter is one of the parameters that may be 
used in selecting which scoring mechanism to use to process a query. The relevance 
parameter value may be determined for each scoring mechanism in a variety of ways. For 
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example, according to one embodiment, the relevance parameter value is determined for each 
scoring mechanism by having a human operator grade the results produced by each scoring 
mechanism for one or more sample queries. For example, after inspecting the search results 
produced by scoring mechanisms A, B and C for fifty sample queries, a human operator may 
assign scoring mechanisms A, B, and C relevance scores of 90, 53 and 82, respectively. 
[0053] The grading by human operators may also be done for each of the scoring 
mechanisms for each of multiple product item categories. The grades compiled from one or 
more human operators are compiled and produce a score for the scoring mechanism for each 
of one ore more product item categories. For example, rather than have a single relevance 
score (e.g. 90) for scoring mechanism A, scoring mechanism A may have a relevance score 
of 70 for queries relating to product category X, a relevance score of 98 for queries relating 
product category Y, and a relevance score of 82 for queries relating to product category Z. 
[0054] The relevance parameter may also be generated automatically. For example, 
relevance for a scoring mechanism may be determined on a predefined training dataset 
including training queries, corresponding training result sets for the training queries, and a 
training product database from which the scoring mechanism chooses results. The relevance 
for the scoring mechanism may then be determined as a function of the number of correct 
and incorrect results that the scoring mechanism selected from the training product database 
(i.e. the relevance parameter value is determined by looking at the overlap between the result 
set the scoring mechanism generates for the training query and the training result set 
corresponding to the training query). 

DETERMINING DENSITY OF SPONSORED PRODUCTS OF SCORING MECHANISM 
[0055] As mentioned above, a density of sponsored product items parameter is one of the 
parameters that may be used in selecting which scoring mechanism to use to process a query. 
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The density of sponsored product items parameter value may be determined for each scoring 
mechanism in a variety of ways. For example, according to one embodiment, the density of 
sponsored product items parameter value may be generated for a scoring mechanism based 
on the percentage of sponsored product items within results that would be produced by using 
the scoring mechanism to process the query. The density of sponsored product items 
parameter value may also be based on potential revenue (e.g. total dollars charged to 
sponsors for inclusion of their product items) for the result set generated by each scoring 
mechanism of said plurality of scoring mechanisms. Any other appropriate density of 
sponsored product items algorithm may also be used. 

DETERMINING USER RETENTION PARAMETER FOR SCORING MECHANISM 
[0056] As mentioned above, a user retention parameter is one of the parameters that may 
be used in selecting which scoring mechanism to use to process a query. The user retention 
parameter value may be determined for each scoring mechanism in a variety of ways. For 
example, according to one embodiment, the user retention parameter value is based on 
estimates of the likelihood that a user that issued a query will find the results useful. It may 
be calculated as the percentage of users that return to a website associated with a scoring 
mechanism within a predetermined time span after receiving results generated by the scoring 
mechanism. Details of two methods for determining user retention are discussed hereafter. 

DETERMINING USER RETENTION BASED ON COOKIES 

[0057] One of the parameter values on which selection of a scoring mechanism from 
among multiple candidate scoring mechanisms may be made is user retention. There are 
general methods described herein for determining user retention. The first uses Internet 
browser cookies and the other is based on user history. 
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[0058] FIG. 3 is a flow diagram that illustrates a process for calculating user retention 
based on cookies. When a user visits a website associated with a search mechanism, a cookie 
is placed on the user's computer (step 310). A browser cookie may be a string that is stored 
on a machine-readable medium associated with the user, a credential identifying the user, or 
any other appropriate identification mechanism. For example, when a user 110 visits a 
website associated with a search mechanism 130, a browser cookie is stored on the user's 
110 computer. 

[0059] Each selection for use of each scoring mechanism for each user is recorded in step 
320. Generally, every selection for use of a particular scoring mechanism from among a 
plurality of scoring mechanisms for use on a query by the user 1 10 is recorded in the 
database 130. The use of the scoring mechanism is associated in the database 130 with the 
user 1 10 by recording it with respect to the browser cookie. The recordation may be 
accomplished in many ways, including adding a record to the log 150 for each selection for 
use of each scoring mechanism. For example, a record is made in a database 140 each time 
each particular user 110 has a particular scoring mechanism selected for use and, in this case, 
the user 1 10 is associated with the browser cookie. 

[0060] In step 330, the cookie is used to determine whether a user has returned to a 
website associated with the search mechanism 130. Generally, a user's 110 return is 
determined based on existence on the user's 1 10 computer of a browser cookie (which was 
stored thereon in step 310). A user 110 will typically be one of multiple users 110 that use a 
search mechanism. Each user 1 10 will have a scoring mechanism selected for each query the 
user submits. All of these scoring mechanism selections and their timings are recorded as 
part of step 330. 
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[0061] The user retention parameter value is calculated for each scoring mechanism 
based on whether each user has returned to a website associated with the search mechanism 
within a predetermined period of time after having the particular scoring mechanism selected 
for use on one the user's 1 10 queries (step 340). The predefined period of time may be any 
appropriate amount of time including one day, two days, one week, two weeks, one month. 
For example, a user retention parameter value is calculated for a particular scoring 
mechanism based on the percentage of users 1 10 that have returned to a website associated 
with the search mechanism 130 within one week of having had the scoring mechanism 
selected for their use. 

[0062] The process for calculating user retention based on cookies enables the generation 
of one of the possible parameter values to be used in selecting a scoring mechanism from 
among multiple candidate scoring mechanisms. 

DETERMINING USER RETENTION BASED ON USER HISTORY 
[0063] A second method of determining user retention utilizes history of users who have 
had selected for their use particular scoring mechanisms. FIG. 4 is a flow diagram that 
illustrates a process for calculating a user retention parameter value based on user history. 
[0064] Referring to FIG. 4, users are logged into a web site associated with a search 
mechanism (step 410). Any appropriate method of logging in a user may be used, including 
having the user provide a user name and password, a credential stored, or other appropriate 
identification information. A user may also log in by typing a universal resource locator 
(URL) into a web browser. For example, a user 110 logs into a website associated with a 
search mechanism 130 using a user name and password. Alternatively, a user may log in by 
running a web service that communicatively couples the user to the search mechanism. 
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[0065] In step 420, each selection for use of each scoring mechanism for one or more 
users is recorded. Generally, every selection for use of a particular scoring mechanism from 
among a plurality of scoring mechanisms for use on a query by the user 1 10 is recorded in 
the database 130. The use of the scoring mechanism is associated in the database 130 with 
the user 1 10 by recording the use with respect to the identifier for the user 1 10. The 
identifier may be any appropriate identifier, such as email address and login ID. The 
recordation may be accomplished by adding a record to the log 150 for each selection for use 
of each scoring mechanism. For example, a record is made in a database 140 each time each 
particular user 1 10 has a particular scoring mechanism selected for use and the user 1 10 is 
identified by login ID. 

[0066] The user history is used to determine whether a user has returned to a website 
associated with the search mechanism (step 430). User activity, as noted above may be 
stored in any appropriate location including the database 140, the log 150, or any other 
appropriate machine-readable medium communicatively coupled to the search mechanism 
130. For example, in the context of FIG. 1, a search mechanism 130 determines that a user 
1 10 has returned to a website associated with the search mechanism 130, based on the history 
for the user 110 stored in a database 140. A user 1 10 will typically be one of multiple users 
110 that use a search mechanism. Each user 110 will have a scoring mechanism selected for 
each query the user submits. All of these scoring mechanism selected and their timings are 
recorded as part of step 430. 

[0067] A user retention parameter value is calculated for each scoring mechanism based 
on whether each user has returned to a website associated with the search mechanism within 
a predetermined period of time after having the particular scoring mechanism selected for use 
on one the user's 1 10 queries (step 440). The period of time may be any appropriate time 



Docket No. 50269-0567 



-18- 



period such as one day, two days, one week, two weeks, or one month. For example, a user 
retention parameter value is calculated for each particular scoring mechanism based on the 
percentage of users 1 10 that have returned to a website associated with a search mechanism 
130 within one day of using the particular scoring mechanism. 
[0068] The process for calculating user retention based on user history enables the 
generation of one of the possible parameter values to be uses in selecting a scoring 
mechanism from among multiple candidate scoring mechanisms. 

HARDWARE OVERVIEW 
[0069] Figure 5 is a block diagram that illustrates a computer system 500 upon which an 
embodiment of the invention may be implemented. Computer system 500 includes a bus 502 
or other communication mechanism for communicating information, and a processor 504 
coupled with bus 502 for processing information. Computer system 500 also includes a main 
memory 506, such as a random access memory (RAM) or other dynamic storage device, 
coupled to bus 502 for storing information and instructions to be executed by processor 504. 
Main memory 506 also may be used for storing temporary variables or other intermediate 
information during execution of instructions to be executed by processor 504. Computer 
system 500 further includes a read only memory (ROM) 508 or other static storage device 
coupled to bus 502 for storing static information and instructions for processor 504. A 
storage device 510, such as a magnetic disk or optical disk, is provided and coupled to bus 
502 for storing information and instructions. 

[0070] Computer system 500 may be coupled via bus 502 to a display 512, such as a 
cathode ray tube (CRT), for displaying information to a computer user. An input device 514, 
including alphanumeric and other keys, is coupled to bus 502 for communicating information 
and command selections to processor 504. Another type of user input device is cursor 
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control 516, such as a mouse, a trackball, or cursor direction keys for communicating 
direction information and command selections to processor 504 and for controlling cursor 
movement on display 512. This input device typically has two degrees of freedom in two 
axes, a first axis (e.g., x) and a second axis (e.g., y), that allows the device to specify 
positions in a plane. 

[0071] The invention is related to the use of computer system 500 for implementing the 
techniques described herein. According to one embodiment of the invention, those 
techniques are performed by computer system 500 in response to processor 504 executing 
one or more sequences of one or more instructions contained in main memory 506. Such 
instructions may be read into main memory 506 from another machine-readable medium, 
such as storage device 510. Execution of the sequences of instructions contained in main 
memory 506 causes processor 504 to perform the process steps described herein. In 
alternative embodiments, hard-wired circuitry may be used in place of or in combination with 
software instructions to implement the invention. Thus, embodiments of the invention are 
not limited to any specific combination of hardware circuitry and software. 
[0072] The term "machine-readable medium" as used herein refers to any medium that 
participates in providing data that causes a machine to operation in a specific fashion. In an 
embodiment implemented using computer system 500, various machine-readable media are 
involved, for example, in providing instructions to processor 504 for execution. Such a 
medium may take many forms, including but not limited to, non-volatile media, volatile 
media, and transmission media. Non- volatile media includes, for example, optical or 
magnetic disks, such as storage device 510. Volatile media includes dynamic memory, such 
as main memory 506. Transmission media includes coaxial cables, copper wire and fiber 
optics, including the wires that comprise bus 502. Transmission media may also take the 
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form of acoustic or light waves, such as those generated during radio-wave and infra-red data 
communications. 

[0073] Common forms of machine-readable media include, for example, a floppy disk, a 
flexible disk, hard disk, magnetic tape, or any other magnetic medium, a CD-ROM, any other 
optical medium, punchcards, papertape, any other physical medium with patterns of holes, a 
RAM, a PROM, and EPROM, a FLASH-EPROM, any other memory chip or cartridge, a 
carrier wave as described hereinafter, or any other medium from which a computer may read. 
[0074] Various forms of machine-readable media may be involved in carrying one or 
more sequences of one or more instructions to processor 504 for execution. For example, the 
instructions may initially be carried on a magnetic disk of a remote computer. The remote 
computer may load the instructions into its dynamic memory and send the instructions over a 
telephone line using a modem. A modem local to computer system 500 may receive the data 
on the telephone line and use an infra-red transmitter to convert the data to an infra-red 
signal. An infra-red detector may receive the data carried in the infra-red signal and 
appropriate circuitry may place the data on bus 502. Bus 502 carries the data to main 
memory 506, from which processor 504 retrieves and executes the instructions. The 
instructions received by main memory 506 may optionally be stored on storage device 510 
either before or after execution by processor 504. 

[0075] Computer system 500 also includes a communication interface 518 coupled to bus 
502. Communication interface 518 provides a two-way data communication coupling to a 
network link 520 that is connected to a local network 522. For example, communication 
interface 518 may be an integrated services digital network (ISDN) card or a modem to 
provide a data communication connection to a corresponding type of telephone line. As 
another example, communication interface 518 may be a local area network (LAN) card to 
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provide a data communication connection to a compatible LAN. Wireless links may also be 
implemented. In any such implementation, communication interface 518 sends and receives 
electrical, electromagnetic or optical signals that carry digital data streams representing 
various types of information. 

[0076] Network link 520 typically provides data communication through one or more 
networks to other data devices. For example, network link 520 may provide a connection 
through local network 522 to a host computer 524 or to data equipment operated by an 
Internet Service Provider (ISP) 526. ISP 526 in turn provides data communication services 
through the world wide packet data communication network now commonly referred to as 
the "Internet" 528. Local network 522 and Internet 528 both use electrical, electromagnetic 
or optical signals that carry digital data streams. The signals through the various networks 
and the signals on network link 520 and through communication interface 518, which carry 
the digital data to and from computer system 500, are exemplary forms of carrier waves 
transporting the information. 

[0077] Computer system 500 may send messages and receive data, including program 
code, through the network(s), network link 520 and communication interface 518. In the 
Internet example, a server 530 may transmit a requested code for an application program 
through Internet 528, ISP 526, local network 522 and communication interface 518. 
[0078] The received code may be executed by processor 504 as it is received, and/or 
stored in storage device 510, or other non- volatile storage for later execution. In this manner, 
computer system 500 may obtain application code in the form of a carrier wave. 

EXTENSIONS AND ALTERNATIVES 
[0079] In the foregoing specification, embodiments of the invention have been described 
with reference to numerous specific details that may vary from implementation to 
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implementation. Thus, the sole and exclusive indicator of what is the invention, and is 
intended by the applicants to be the invention, is the set of claims that issue from this 
application, in the specific form in which such claims issue, including any subsequent 
correction. Any definitions expressly set forth herein for terms contained in such claims shall 
govern the meaning of such terms as used in the claims. Hence, no limitation, element, 
property, feature, advantage or attribute that is not expressly recited in a claim should limit 
the scope of such claim in any way. The specification and drawings are, accordingly, to be 
regarded in an illustrative rather than a restrictive sense. 
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